Mobile imaging device as navigator

ABSTRACT

Embodiments of the invention are directed to obtaining information based on directional orientation of a mobile imaging device, such as a camera phone. Visual information is gathered by the camera and used to determine a directional orientation of the camera, to search for content based on the direction, to manipulate 3D virtual images of a surrounding area, and to otherwise use the directional information. Direction and motion can be determined by analyzing a sequence of images. Distance from a current location, inputted search parameters, and other criteria can be used to expand or filter content that is tagged with such criteria. Search results with distance indicators can be overlaid on a map or a camera feed. Various content can be displayed for a current direction, or desired content, such as a business location, can be displayed only when the camera is oriented toward the desired content.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a Continuation Patent Application of U.S.patent application Ser. No. 13/252,003 entitled “Mobile Imaging DeviceAs Navigator,” filed on Oct. 3, 2011 which is a Continuation PatentApplication of U.S. patent application Ser. No. 12/143,527 entitled“Mobile Imaging Device As Navigator,” filed on Jun. 20, 2008, thebenefit of the earlier filing date of which is hereby claimed under 35U.S.C. §120 and which is further incorporated herein by reference in itsentirety.

FIELD OF ART

The present invention is directed to obtaining information based onorientation of a mobile imaging device, and more particularly, toenabling a user to orient directionally in real and/or virtual worlds byorienting the mobile imaging device to capture visual cues fordirectional guidance.

BACKGROUND

Three-dimensional (3D) vector maps are used in global positioning system(GPS) mapping systems such as those by TomTom International BV andGarmin Ltd. Such maps may give a perspective view of a locality from anoverhead “birds-eye” view or from a ground-level view. A perspectiveview enables a user to gauge how far away one might be from a location.Existing GPS vector map systems often come pre-programmed with certainlocal destinations, such as restaurants and coffee shops. A driver canautomatically scroll through these maps as a function of linear motionthrough space as tracked by GPS, but map information is generally basedon the direction of motion. Also, the scale of this linear motion, andthe corresponding navigation on the map, is typically tuned for drivingrather than foot movement.

Similarly, current mobile search systems, such as those used by personaldata assistants (PDAs), typically use a 2D birdseye map view to displaylocal results in a radius around a current location of the mobiledevice, rather than in a direction that a user is facing. Thus, current2D and 3D mapping systems generally provide search information in adirection of travel, or in a radius about a location. But these systemsgenerally can not determine an orientation in which a driver or user isfacing. For example, a car may move in reverse, but the driver may befacing forward or sideways. It is with respect to these considerationsand others that the present invention is directed.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive embodiments of the present invention aredescribed with reference to the following drawings. In the drawings,like reference numerals refer to like parts throughout the variousfigures unless otherwise specified.

For a better understanding of the present invention, reference will bemade to the following Detailed Description Of The Embodiments, which isto be read in association with the accompanying drawings, wherein:

FIG. 1 illustrates a diagram of one embodiment of an exemplary system inwhich the invention may be practiced;

FIG. 2 shows a schematic diagram of one embodiment of an exemplarymobile device;

FIG. 3 illustrates a schematic diagram of one embodiment of an exemplarynetwork device; and

FIG. 4 illustrates a flow chart of example logic for determining searchresults based on a direction in which the mobile device is pointed.

DETAILED DESCRIPTION OF EMBODIMENTS

The present invention now will be described more fully hereinafter withreference to the accompanying drawings, which form a part hereof, andwhich show, by way of illustration, specific exemplary embodiments bywhich the invention may be practiced. This invention may, however, beembodied in many different forms and should not be construed as limitedto the embodiments set forth herein; rather, these embodiments areprovided so that this disclosure will be thorough and complete, and willfully convey the scope of the invention to those skilled in the art.Among other things, the present invention may be embodied as methods ordevices. Accordingly, the present invention may take the form of anentirely hardware embodiment, an entirely software embodiment or anembodiment combining software and hardware aspects. The followingdetailed description is, therefore, not to be taken in a limiting sense.

Throughout the specification and claims, the following terms take themeanings explicitly associated herein, unless the context clearlydictates otherwise. The phrase “in one embodiment” as used herein doesnot necessarily refer to the same embodiment, though it may.Furthermore, the phrase “in another embodiment” as used herein does notnecessarily refer to a different embodiment, although it may. Thus, asdescribed below, various embodiments of the invention may be readilycombined, without departing from the scope or spirit of the invention.

In addition, as used herein, the term “or” is an inclusive “or”operator, and is equivalent to the term “and/or,” unless the contextclearly dictates otherwise. The term “based on” is not exclusive andallows for being based on additional factors not described, unless thecontext clearly dictates otherwise. In addition, throughout thespecification, the meaning of “a,” “an,” and “the” include pluralreferences. The meaning of “in” includes “in” and “on.”

Briefly stated, embodiments of the invention are directed to obtaininginformation based on directional orientation of a mobile imaging device.Embodiments enable users to directionally orient themselves by orientingthe mobile imaging device to capture visual cues for directionalguidance. The visual clues are used to determine a directionalorientation of the mobile device, perform searches based on theorientation, perform orientational manipulation of three-dimensionalvirtual worlds, and the like. Directional orientation information mayalso be combined with distance information to expand or filterinformation.

In one embodiment, a mobile map application is used with a cell phonehandset camera for improving user navigation and finding identifiedlocations of interest. A handset camera is used for trackingorientational motion and small scale linear motion. Once, a user'slocation has been pinpointed (e.g., via GPS, by user input, or thelike), the user can gather information on directional orientation byturning in a circle and having the handset camera capture visual/opticalcues, such as intersections, store fronts, signs, etc. The visual cluesare associated with predefined information of a 3D vector map fordirectional guidance. In another embodiment, optical input from thehandset camera (e.g., tracking patterns in the background to discernspeed and direction of the handset camera) is used to dynamicallynavigate around 3D vectors maps. Predefined vector map information maybe sent to the handset, so that it is easier to find identifiedlocations. For example, streets, buildings, landmarks, or other pointsof interest may be several blocks away and obscured by tall buildingsrelative to where the user is presently situated. Similarly, localmerchant information from one or more sources may also be provided onthe vector map. A user may manage the various types of information, andthe amount of information by indicating an orientation without requiringlinear motion to identify a direction. With orientation data andmultiple types of content information, embodiments of the inventionenable more relevant mobile search results, mobile navigation, and otherinformation.

Directional orientation information enables searching “infinitely” farin a direction, and each search result may be marked with a distancerelative to where the user's current position. Indications of distancemay be a proportionally smaller sized icon, a lower rank on a resultslist, a degree of transparency, or other indicator. For example, a usermay be stopped at an intersection, trying to determine which way to turnto find the nearest (or a preferred company's) gas station. Withouthaving to travel down each road, and without having to scroll or changea zoom level on a virtual map, the user can simply point a mobile devicein the direction of each road and receive results with distanceindicators.

One embodiment also utilizes information regarding orientational motionof the mobile device. Initially, a mobile device establishes itsexisting physical position via GPS, user input, image analysis of acurrent scene, or the like. The mobile device also uses its imagingcomponent to establish its initial orientational direction. Theorientation may be determined by comparing an image with a knowndirectional image. This comparison may be performed on an image capturedand stored by the mobile device, on an image in a viewfinder that issuperimposed with an outline of a known object, on a combination ofimages, or using other image processing techniques. Alternate, oradditional techniques may include analyzing sun position relative todate and time, receiving instructions to orient the mobile device in aspecified direction and clicking to confirm, analyzing text or symbolson signs at known locations, or the like.

Once an initial orientation is established, some embodiments involvedetecting a change in orientation and/or detecting a new orientationafter motion. For example, the mobile device may detect a new directionin which the device is pointing during, or after the device is moved ina circle around the user. At the initial orientation, during the motion,and/or at the new orientation, the mobile device performs may displayand move a map based on the orientations and change in orientation. Themobile device may also perform one or more content searches or otherwiseobtains content information regarding points of interest in thedirection(s) that the mobile device is pointed. The content informationmay be superimposed on the displayed map, and may move and/or changebased on the orientations and change in orientation. Additional detailof example embodiments is described below with regard to the figures.

Illustrative Operating Environment

FIG. 1 illustrates one embodiment of an environment in which the presentinvention may operate. However, not all of these components may berequired to practice the invention, and variations in the arrangementand type of the components may be made without departing from the spiritor scope of the invention.

As shown in the figure, system 100 includes client devices 102-104,network 105, and a server 106. Network 105 is in communication with andenables communication between each of client devices 102-104, and server106.

Client devices 102-104 may include virtually any computing devicecapable of receiving and sending a message over a network, such asnetwork 105, to and from another computing device, such as contentserver 108, each other, and the like. The set of such devices mayinclude devices that typically connect using a wireless communicationsmedium such as cell phones, smart phones, pagers, walkie talkies, radiofrequency (RF) devices, infrared (IR) devices, CBs, integrated devicescombining one or more of the preceding devices, or virtually any mobiledevice, and the like. Similarly, client devices 102-104 may be anydevice that is capable of connecting using a wired or wirelesscommunication medium such as a personal digital assistant (PDA), pocketPC, wearable computer, and any other device that is equipped tocommunicate over a wired and/or wireless communication medium. The setof such devices may also include devices that typically connect using awired communications medium such as personal computers, multiprocessorsystems, microprocessor-based or programmable consumer electronics,network PCs, and the like.

Each client device within client devices 102-104 may include a browserapplication that is configured to send, receive, and display web pages,and the like. The browser application may be configured to receive anddisplay graphics, text, multimedia, and the like, employing virtuallyany web based language, including, but not limited to StandardGeneralized Markup Language (SMGL), such as HyperText Markup Language(HTML), extensible markup language (XML), a Handheld Device MarkupLanguage (HDML), such as Wireless Markup Language (WML), WMLScript,JavaScript, and the like. Client devices 102-104 may further include amessaging application configured to send and/or receive a messageto/from another computing device employing another mechanism, including,but not limited to instant messaging (IM), email, Short Message Service(SMS), Multimedia Message Service (MMS), internet relay chat (IRC),mIRC, Jabber, and the like.

Network 105 is configured to couple one computing device to anothercomputing device to enable them to communicate. Network 105 is enabledto employ any form of computer readable media for communicatinginformation from one electronic device to another. Also, network 105 mayinclude a wireless interface, and/or a wired interface, such as theInternet, in addition to local area networks (LANs), wide area networks(WANs), direct connections, such as through a universal serial bus (USB)port, other forms of computer-readable media, or any combinationthereof. On an interconnected set of LANs, including those based ondiffering architectures and protocols, a router acts as a link betweenLANs, enabling messages to be sent from one to another. Also,communication links within LANs typically include twisted wire pair orcoaxial cable, while communication links between networks may utilizeanalog telephone lines, full or fractional dedicated digital linesincluding T1, T2, T3, and T4, Digital Signal level 3 (DS3), OpticalCarrier 3 (OC3), OC12, OC48, Asynchronous Transfer Mode (ATM),Integrated Services Digital Networks (ISDNs), Digital Subscriber Lines(DSLs), wireless links including satellite links, or othercommunications links known to those skilled in the art. Furthermore,remote computers and other related electronic devices could be remotelyconnected to either LANs or WANs via a modem and temporary telephonelink. Network 105 is constructed for use with various communicationprotocols and technologies, including transmission controlprotocol/internet protocol (TCP/IP), user datagram protocol (UDP), awireless application protocol (WAP), global system for mobilecommunications (GSM), code division multiple access (CDMA), timedivision multiple access (TDMA), general packet radio service (GPRS),ultra wide band (UWB), IEEE 802.16 Worldwide Interoperability forMicrowave Access (WiMax), and the like. In essence, network 105 includesany communication method by which information may travel between clientdevices 102-104, and/or server 106.

The media used to transmit information in communication links asdescribed above generally includes any media that can be accessed by acomputing device. Computer-readable media may include computer storagemedia, wired and wireless communication media, or any combinationthereof Additionally, computer-readable media typically embodiescomputer-readable instructions, data structures, program modules, orother data. Such data can be communicated through communication media ina modulated data signal such as a carrier wave, data signal, or othertransport mechanism and includes any information delivery media. Theterms “modulated data signal,” and “carrier-wave signal” includes asignal that has one or more of its characteristics set or changed insuch a manner as to encode information, instructions, data, and thelike, in the signal. By way of example, communication media includeswireless media such as fluids or space for acoustic, RF, infrared, andother wireless signals, and wired media such as twisted pair, coaxialcable, fiber optics, wave guides, and other wired media.

Server 106 may comprise multiple computing devices or a single computingdevice.

Server 106 may provide image analysis services, such as determininghistograms, performing OCR, comparing images to previously storedimages, determining information about images, performing databaseoperations, performing searches for additional information, storinginformation about images, tracking user behaviors, or the like. Server106 may also provide content and/or other services such as web sites,online journals (e.g., blogs), photos, reviews, online services such asmessaging, search, news, shopping, advertising, and/or the like. Server106 may further provide administrative services, such as creation,modification, and management of relationships between network resources,such as web pages, or the like. Briefly, server 106 may include anycomputing device capable of connecting to network 105 and may manageservices for a network user, such as a user of at least one of clientdevices 102-104. Devices that may operate as server 106 includededicated server devices, personal computers, desktop computers,multiprocessor systems, microprocessor-based or programmable consumerelectronics, network PCs, and the like. Server 106 and/or any of clients102-104 may be implemented on one or more computing devices, such as aclient described with regard to FIG. 2.

Illustrative Client Device

FIG. 2 shows an exemplary client device 200, according to one embodimentof the invention for use as a reference data collector device. In oneembodiment, client device 200 is a mobile device, such as a laptopcomputer. Another example of a mobile device includes a PDA or acellular telephone that is arranged to send and receive voicecommunications and messages such as SMS messages via one or morewireless communication interfaces. Oftentimes, mobile electronic deviceswill be capable of personal communication by connecting to one or morewireless networks, connecting to multiple nodes of a single wirelessnetwork, communicating over one or more channels to one or morenetworks, or otherwise engaging in one or more communication sessions.Generally, client device 200 may comprise any mobile or stationaryelectronic device. Such devices include laptops, palmtops, PDAs,handheld computers, cellular telephones, smart phones, pagers, radiofrequency (RF) devices, infrared (IR) devices, integrated devicescombining one or more of the preceding devices, and the like. Clientdevice 200 may also comprise other electronic devices such as personalcomputers, multiprocessor systems, microprocessor-based or programmableconsumer electronics, network PCs, wearable computers, and the like.

Client device 200 may include many more, or fewer, components than thoseshown in FIG. 2. However, the components shown are sufficient todisclose an illustrative embodiment for practicing the presentinvention. As shown in the figure, client device 200 includes aprocessing unit 222 in communication with a mass memory 230 via a bus224.

Mass memory 230 includes a RAM 232, a ROM 234, and other storage means.Mass memory 230 illustrates another example of computer storage mediafor storage of information such as computer readable instructions, datastructures, program modules or other data. Mass memory 230 stores abasic input/output system (“BIOS”) 240 for controlling low-leveloperation of client device 200. The mass memory also stores an operatingsystem 241 for controlling the operation of client device 200. It willbe appreciated that this component may include a general purposeoperating system such as a version of Windows®, UNIX, or LINUX®, or aspecialized mobile communication operating system such as WindowsMobile™, the Symbian® operating system, or the like. The operatingsystem may include, or interface with a Java® virtual machine modulethat enables control of hardware components and/or operating systemoperations via Java application programs.

Memory 230 further includes one or more data storage units 242, whichcan be utilized by client device 200 to store, among other things,programs 244 and/or other data. Programs 244 may include computerexecutable instructions which, when executed by client device 200,transmit, receive, render, and/or otherwise process markup pages such asHTML pages, XML pages, WAP pages (sometimes referred to as WAP cards),and the like. Accordingly, programs 244 may include a browser program ofcomputer executable instructions, which may be run under control ofoperating system 241 to enable and manage requesting, receiving, andrendering markup pages and messages (e.g., HTTP, TCP/IP, SMS, MMS, IM,email, and/or other messages), audio, video, and enabletelecommunication with another user of another client device. Otherexamples of application programs include calendars, contact managers,task managers, transcoders, database programs, word processing programs,spreadsheet programs, games, and so forth.

In addition, mass memory 230 stores an image handling module 246 and amapping module 248, which may be run as external modules under controlof operating system 241, as plug-in modules within a browser, within aweb page, or via another configuration. Image handling module 246 mayperform initial processing or complete processing of image data, contextinformation, user requests for image data, and other data related toimages acquired or received by the mobile device. Mapping module 248 maymanage map information for display, perspective, motion, integrationwith images or content, and other map operations.

Client device 200 also includes a power supply 226, one or more wirelessinterfaces 250, an audio interface 252, a display 254, a keypad 256, anilluminator 258, an optional data capture module 259, an input/outputinterface 260, an optional haptic interface 262, an optional globalpositioning systems (GPS) receiver 264, and an optional sensor thatsenses motion, direction, and/or orientation of the client device. Powersupply 226 provides power to client device 200. A rechargeable ornon-rechargeable battery may be used to provide power. The power mayalso be provided by an external power source, such as an AC adapter or apowered docking cradle that supplements and/or recharges a battery.

Client device 200 may optionally communicate with a base station, ordirectly with another client device. Wireless interface 250 includescircuitry for coupling client device 200 to one or more wirelessnetworks, and is constructed for use with one or more communicationprotocols and technologies including, but not limited to, TCP/IP, UDP,GSM, CDMA, TDMA, SMS, GPRS, WAP, UWB, IEEE 802.16 (WiMax), and the like.

Audio interface 252 is arranged to produce and/or receive audio signalssuch as the sound of a human voice, music, and the like. For example,audio interface 252 may be coupled to a speaker and microphone (notshown) to enable telecommunication with others and/or generate an audioacknowledgement for some action. Display 254 may be a liquid crystaldisplay (LCD), gas plasma, light emitting diode (LED), or any other typeof display used with a client device. Display 254 may also include atouch sensitive screen arranged to receive input from an object such asa stylus or a digit from a human hand.

Keypad 256 may comprise any input device arranged to receive input froma user. For example, keypad 256 may include a keyboard, a push buttonnumeric dial, or the like. Keypad 256 may also include command buttonsthat are associated with selecting and performing changeable processes.Illuminator 258 may provide a status indication and/or provide light.Illuminator 258 may remain active for specific periods of time or inresponse to events. For example, when illuminator 258 is active, it maybacklight the buttons on keypad 256 and stay on while the client deviceis powered. Also, illuminator 258 may backlight these buttons in variouspatterns when particular actions are performed, such as dialing anotherclient device. Illuminator 258 may also cause light sources positionedwithin a transparent or translucent case of the mobile device toilluminate in response to actions. Illuminator 258 may further be usedas a flash for image capture. A data capture module 259, such as acamera, may be included in client device 200. The client device mayobtain images, video, temperature, pressure, or other data.

Client device 200 also comprises input/output interface 260 forcommunicating with external devices, such as a headset, or other inputor output devices not shown in FIG. 2. Input/output interface 260 canutilize one or more communication technologies, such as USB, infrared,Bluetooth™, and the like. Optional haptic interface 262 is arranged toprovide tactile feedback to a user of the client device. For example,the haptic interface may be employed to vibrate client device 200 in aparticular way when another user of a client device is calling.

Optional GPS transceiver 264 can determine the physical coordinates ofclient device 200 on the surface of the Earth, which typically outputs alocation as latitude and longitude values. GPS transceiver 264 can alsoemploy other geo-positioning mechanisms, including, but not limited to,triangulation, assisted GPS (AGPS), Enhanced Observed Time Difference(E-OTD), cell identifier (CI), service area identifier (SAI), enhancedtiming advance (ETA), base station subsystem (BSS), or the like, tofurther determine the physical location of client device 200 on thesurface of the Earth. It is understood that under different conditions,GPS transceiver 264 can determine a physical location within millimetersfor client device 200; and in other cases, the determined physicallocation may be less precise, such as within a meter or significantlygreater distances.

Optional sensor 266 can detect a direction, orientation, motion, orother aspect of the mobile device. The sensor may read staticinformation or dynamic information, such as a change to any aspect ofthe mobile device. Sensor 266 may comprise an accelerometer, agyroscope, a compass, or other sensor that detects motion, direction, orthe like. The sensor may include, or be coupled to transducingcomponents, signal processing components, and/or other components.

Illustrative Network Device

FIG. 3 shows one embodiment of a network device, according to oneembodiment of the invention. Network device 300 may include many more,or fewer, components than those shown. The components shown, however,are sufficient to disclose an illustrative embodiment for practicing theinvention. Network device 300 may represent, for example, server 106 oranother client device of FIG. 1. For example purposes, network device300 will be described as a server device.

As shown in the figure, server device 300 includes a processing unit 322in communication with a mass memory 330 via a bus 324. Mass memory 330generally includes a RAM 332, a ROM 334, and other storage means. Massmemory 330 illustrates a type of computer-readable media, namelycomputer storage media. Computer storage media may include volatile andnonvolatile, removable and non-removable media implemented in any methodor technology for storage of information such as computer readableinstructions, data structures, program modules or other data. Otherexamples of computer storage media include EEPROM, flash memory or othersemiconductor memory technology, CD-ROM, digital versatile disks (DVD)or other optical storage, magnetic cassettes, magnetic tape, magneticdisk storage or other magnetic storage devices, or any other mediumwhich can be used to store the desired information and which can beaccessed by a computing device.

Mass memory 330 stores a basic input/output system (“BIOS”) 340 forcontrolling low-level operation of server device 300. The mass memoryalso stores an operating system 341 for controlling the operation ofserver device 300. It will be appreciated that this component mayinclude a general purpose operating system such as a version of Windows,UNIX, LINUX, Solaris, or the like. The operating system may alsoinclude, or interface with a Java virtual machine module that enablescontrol of hardware components and/or operating system operations viaJava application programs.

Mass memory 330 further includes one or more data storage units 342,which can be utilized by server device 300 to store, among other things,programs 344 and/or other data. Programs 344 may include computerexecutable instructions which can be executed by server device 300 toimplement a markup handler application, such as an HTTP handlerapplication for transmitting, receiving, and otherwise processing HTTPcommunications, a WAP handler application for transmitting, receiving,and otherwise processing WAP communications, and the like. Similarly,programs 344 can include a secure socket layer (SSL) handler applicationfor handling secure connections, such as initiating communication withan external application in a secure fashion. Other examples ofapplication programs include content management applications, messagingapplications, schedulers, calendars, web services, transcoders, databaseprograms, word processing programs, spreadsheet programs, and so forth.Accordingly, programs 344 can process images, audio, video, or markuppages, enable telecommunication with another user of another electronicdevice, and/or other services.

In addition, mass memory 330 stores an image processing module 346.Image processing module 346 may include computer executableinstructions, which may be run under control of operating system 341 toanalyze images, compare images, determine context information, accessinformation associated with a location, evaluate user behavior data,determined search terms, or perform other processes related to images.In one embodiment, image processing module 346 generally communicateswith an image handling module on a client device. Mass memory 330 mayalso include a search module 348, which may be part of, or incommunication with image processing module 346. Search module 348 maycomprise a search engine that generally performs data searches, such ascontent searches, searches for links to network resources, databasesearches, or the like.

Server device 300 also includes an input/output interface 360 forcommunicating with input/output devices such as a keyboard, mouse,wheel, joy stick, rocker switches, keypad, printer, scanner, and/orother input devices not specifically shown in FIG. 3. A user of serverdevice 300 can use input/output devices to interact with a userinterface that may be separate or integrated with operating system 341,programs 344, and/or other modules. Interaction with the user interfaceincludes visual interaction via a display, and a video display adapter354.

Server device 300 may include a removable media drive 352 and/or apermanent media drive 354 for computer-readable storage media. Removablemedia drive 352 can comprise one or more of an optical disc drive, afloppy disk drive, and/or a tape drive. Permanent or removable storagemedia may include volatile, nonvolatile, removable, and non-removablemedia implemented in any method or technology for storage ofinformation, such as computer readable instructions, data structures,program modules, or other data. Examples of computer storage mediainclude a CD-ROM 355, digital versatile disks (DVD) or other opticalstorage, magnetic cassettes, magnetic tape, magnetic disk storage orother magnetic storage devices, RAM, ROM, EEPROM, flash memory or othermemory technology, or any other medium which can be used to store thedesired information and which can be accessed by a computing device.

Via a network communication interface unit 350, server device 300 cancommunicate with a wide area network such as the Internet, a local areanetwork, a wired telephone network, a cellular telephone network, orsome other communications network, such as network 105 in FIG. 1.Network communication interface unit 350 is sometimes known as atransceiver, transceiving device, network interface card (NIC), and thelike.

Illustrative Logic

FIG. 4 illustrates a flow chart of example logic for determining searchresults based on a direction in which the mobile device is pointed. Inthis example embodiment, processing is primarily performed on a clientmobile device. But at least some operations may be performed with aserver device over a network.

At an operation 402, the current physical geographic position of themobile device is determined. The current location can be established bya built-in GPS component, a radio frequency identification (RFID) tag orreader at a known location, or other position component. Alternatively,the user may tap, click, or otherwise indicate a point on a displayedmap, which can return a latitude, longitude, altitude, street address,tax parcel number, or other location information. As anotheralternative, the user may simple input location information, such as anaddress, an intersection, a building name, a landmark name, or otherlocation information.

In another embodiment, a camera of the mobile device can capture animage, which is then analyzed relative to known location information.For example, the camera may capture an image of a street sign, abillboard, a store front, a building, or other scene. Text, symbols,landmarks, or other information may be analyzed from the image todetermine a location. Further details of image analysis and locationdetection are described in co-owned U.S. patent application Ser. No.12/121,259, filed on May 22, 2008, and titled “Data Access Based OnContent Of Image Recorded By A Mobile Device,” the entire contents ofwhich are hereby incorporated by reference.

At an operation 404, the mobile device is used to establish an initialorientational direction in which the mobile device is pointing. Thisinitial direction can be established in a number of ways, and may beused to synchronize the mobile device with a mapping module or otherservice. For example, based on the determined initial position, themobile device may display a message instructing the user to point themobile device along a known street, toward a known physical object,toward a known compass direction, or other direction that a user caneasily identify. The mobile device can also display a 2D or 3D map ofthe area around the determined initial position, and the map can displayan arrow, a highlighted icon, or other indicator of the street, object,or other directional item at which the mobile device instructed the userto point the mobile device. This displayed map may help the user orientthe mobile device to point in the instructed direction. The mobiledevice may further instruct the user to press a button on the mobiledevice, indicating that the user has oriented the mobile device asinstructed.

In another embodiment, the mobile device may use a 3D first personperspective map of the area around the initial determined position. Morespecifically, the 3D map may show a 3D graphic scene of buildings,terrain, or other rendered objects to which the user should point themobile device. The 3D graphic scene may be shown as a semi-transparentor outline overlay to the camera feed of the mobile device. As the usermoves the camera lens of the mobile device to different directions, theuser can “recognize” when the image in the camera finder “matches” the3D map shown. Similarly, the mobile device may continuously scan thecamera feed and perform continuous image recognition for certainpatterns, compared to a 3D perspective overlay. If a match is detected,the mobile device can automatically discern when the user is pointingthe camera lens in the right direction.

In a further embodiment, the mobile device may use the GPS module todetect that the device is moving in a certain direction, and instructthe user to point the mobile device in the direction that the user istraveling (e.g., by car, by public transportation, by walking, etc.).Again, the mobile device may instruct the user to press a button toindicate that the mobile device is oriented in the instructed direction.In yet another embodiment, the mobile device may instruct the user topoint the mobile device camera lens in the direction of the sun. Afterthe user presses a button, the mobile device can obtain informationabout the current time of day, and use the determined location tocalculate the direction in which the mobile device is pointing. Inaddition, or alternatively, the mobile device may use image analysis toevaluate shadows. In a further embodiment, the mobile device may includea compass. The mobile device may read a direction from the compass whenthe user presses a button. The button may be a general physical button(e.g., an enter button), may be a virtual button displayed on a screen,may be a dedicated “direction” button, or other form of button.

Based on the initial direction, the mobile device can access contentinformation relevant to the initial direction. In another embodiment,the initial direction may be used as a reference direction to determinea change in direction.

In this embodiment, at an operation 406, the mobile device determines achange in direction. For example, the mobile device may detect a neworientational direction during or after the mobile device is moved in acircle around the initial position. The mobile device may analyze eachframe of multiple images taken by the mobile device during the motion,to determine a shift in the images. This shift indicates an amount anddirection of motion, so the mobile device can calculate a new direction.In this embodiment, the mobile device camera continuously snaps a streamof pictures, and analyzes them for patterns, such as lines, blocks, etc.The mobile device then evaluates the movement of these patterns, so thatif the user moves the mobile device in any direction (or orientation)relative to the background, the mobile device can determine the relativemotion. The mobile device may also revise a map display or otherdisplayed content by a similar amount (which may actually be in theopposite direction). The mobile device can detected the shift betweensuccessive patters, or between the first pattern of the first capturedimage and the last pattern of the last captured image. This gives theimpression that the displayed map is fixed in space towards the physicalworld, while the mobile device appears to pan over this map.

In another embodiment, the mobile device may include one or moreaccelerometers, gyroscopes, compasses, strain gauges, or othercomponents for detecting motion, direction, or other properties. Forinstance, three accelerometers may be used to detect motion in threedimensions. Motion relative to the initial direction can be used todetermine a new direction.

While the mobile device is moving, or after the mobile device is held ina new direction, the mobile device obtains content informationassociated with the current direction, at an operation 408. The mobiledevice accesses content information stored on the mobile device and/orat a remote server. The mobile device may access a database or submitsearch requests based on the current location and direction information.The user may also enter keywords or other search parameters. Forexample, the user may enter an address, business name, product type, orother search term that the user may be interested in finding in thedirection that the mobile device is currently pointing. The mobiledevice or a search server can perform the search or may filter searchresults based on the entered search term, current direction, and currentposition. Similarly, the user may enter a maximum distance, or thesearch system may use a default maximum distance from the currentposition. Results can be limited to those that are within the specifieddistance, or within a range around the specified distance.

The content information is tagged with geographic position information.The search system can determine a range of geographic positions relativeto the current position and direction of the mobile device. The searchsystem can then determine content that is tagged with geographicpositions within the determined range. The results can be furtherdetermined based on keywords or other parameters. For example, themobile device user may have given permission for the search system oranother service to track the movement, network interactions, purchases,game activities, or other uses of the mobile device. This informationmay be compiled in a profile associated with the mobile device.Aggregate information may also be compiled and statistically analyzedfor the behaviors of a number of other devices or users. The searchsystem may use the profile information and/or aggregate information tofurther refine the search results to information that may be morerelevant to the mobile device user.

In another example, in the displayed map on the mobile device, the usermay have selected a tab that indicates a certain type of information,such as restaurants. A corresponding search term for the selected tabmay be submitted with the position, direction, distance, and keywords tolimit the results to restaurants in the current direction.

Similarly, the user may select a tab or otherwise indicate that the userdesires information on other users that are currently in the directionthat the mobile device is pointing. For example, users may be using amessaging system, logged into a portal service with a user ID, using asocial networking system that can manage relationships of people who areseparated by multiple degrees of relationships, or otherwise providingcurrent location information that is accessible to the search system. Inone instance, two users may be trying to locate each other along astreet, somewhere in a building, around a very loud crowded room, or inanother environment. A selected “people” tab, an entered name, anentered user ID, or other related search term can be used to limit theresults to those associated with the search term, current direction, andcurrent position. Optional presence information can also be used by thesearch system to determine final results. For example, a private usermay indicate that they want to remain “invisible” to other users who maybe pointing their mobile devices in the direction of the private user.Alternatively, a group user may indicate that they want their presenceto be known only to users in the group user's a contact list.

While determining search results, the search system can also calculatedistances between each search result and the current position of thesearching mobile device. Similarly, the search system may calculate anoffset between each search result and the current direction that themobile device is pointing. The offset may be an angular offset, a linearoffset from a direction line, or other offset.

At an operation 410, the distance information and final results areprovided to the mobile device and displayed by the mobile device. Themobile device may display the search results as a simple list or in agraphical manner. For example, the mobile device may display the list inorder based on distance from the current position of the mobile device.The distance may be specified next to each search result. Similarly, themobile device may indicate an offset next to each search result. Thesearch results may include an icon or other graphic that indicates atype of information. For example, search results for another user may beindicated by an avatar. Search results for a fuel station may beindicated by a fuel symbol. Similarly, colors, fonts, or otherindicators may be used to distinguish search results.

In another embodiment, the mobile device superimposes the search resultsonto a graphic display. For example, the mobile device may superimposeicons, text, or other search result information onto a displayed 3Dvector map. Portions of the search results may be displayed based on auser selected tab, entered keywords, predefined preferences, presenceparameters, past behavior, aggregate behavior, or other filter. Somesearch results may not fall within the current field of the mobiledevice display, and may not be displayed until the view is modified orthe mobile device is pointing in the corresponding physical direction ofthat search result. Alternatively, the actual search results may bedynamically updated as the mobile device is moved around.

In one embodiment, the results are displayed on a 3D perspective viewthat is simply the viewfinder feed from the mobile device camera. Inanother embodiment the search results are superimposed on a 3Dperspective map that may appear fixed, such that the display appears topan over the vector map as the mobile device is moved. In oneembodiment, a semitransparent 3D perspective map outlines the elementssuch as streets, on top of a background image.

Search results that are in the close perimeter of the mobile device'scurrent location, but are not in a current field of view, or beyond athreshold distance, can be cached for quick rendering while moving themobile device. In addition, or alternatively, more and more searchresults may be displayed further and further from the current locationof the device, if the device is kept still in a certain direction.Distance can also be indicated by a relative size of search resultindicators to give a clearer perspective on results that are physicallyclose and those that are not. As with a list of results, graphicallydisplayed results may have a distance indicator next to the result icon.

A further embodiment may provide “X-Ray” search results. This embodimentcan give a user the sense of having an “X-Ray” device that can seethrough buildings and terrain to reveal what is behind obstacles in acertain direction. For example, the results may be displayedsemi-transparently, or the obstacles may be displayedsemi-transparently. Corresponding icons or other results indicators maybe displayed for restaurants, fuel stations, shops, user generatedcontent, or other information. This is useful for searching for specificthings when the user already knows roughly what the user is looking for,such as categories of information or a specific string.

In another example, if the user entered an address, the search systemmay provide search results indicating a different direction than thedetermined direction. In that case, the user may move the mobile device,such as in a circle about its current position, until the address resultcomes into view (directly or through an obstacle). For instance, a reddot may be displayed when the mobile device is moved to the direction ofthe entered address. This enables a user to find a direction, to proceedto the desired address.

Similarly, yet another embodiment may provide an “overlay” and“discovery” capability. This embodiment can give a user the sense ofputting a digital, or virtual overlay on top of a physical place, andallow the user to discover content without looking for anything inparticular. For example, user generated content like, such aslocation-tagged images or comments provided by other users, may bedisplayed around the mobile device user's current location. As the usermoves the direction of the mobile device, the location-tagged images,comments, or other content may be displayed. This may be applicable forcommunity services in general.

At an optional operation 412, the mobile device may detect that the useractivated a zoom function on the mobile device. The mobile devicedetermines a corresponding change to the field of view. In oneembodiment, the change in field of view may simply be represented as anextension to the range of distance in the direction that the mobiledevice is pointed. In another embodiment, the change in field of viewmay be represented as an offset change in current position. Aperspective calculation may also determine a change in angular or linearoffset from the direction line.

At an optional operation 414, the display of search results may beupdated to reflect the change in field of view. Alternatively, a newsearch may be requested, and the search results may be updated.

The above specification, examples, and data provide a completedescription of the manufacture and use of the composition of theinvention. For example, the search results may be provided with an audiooutput device, which may be useful while driving or for legally blindusers. In another embodiment, a movable imaging device, such as a webcamera, may be coupled to a computing device, and only the movableimaging device moves position and/or direction. Since many embodimentsof the invention can be made without departing from the spirit and scopeof the invention, the invention resides in the claims hereinafterappended.

What is claimed as new and desired to be protected by Letters Patent is:1. An apparatus comprising a non-transitory computer readable medium,having computer-executable instructions stored thereon, that in responseto execution by a mobile device, cause the mobile device to performoperations, comprising: determining a geographic position of the mobiledevice; determining a change in an orientational direction of the mobiledevice from a first direction to a second direction, the orientationaldirection being determined at least in part using an imaging componentof the mobile device; receiving one or more search terms at the mobiledevice; performing a search request using the received geographicposition, change in orientational direction of the mobile device, andthe received search terms; and displaying results of the search requestat the mobile device.
 2. The apparatus of claim 1, wherein the mobiledevice performs operations, further comprising: receiving merchantinformation that is useable to refine the search request.
 3. Theapparatus of claim 1, wherein one or more of the search terms are basedon a selection at the mobile device of a type of merchant.
 4. Theapparatus of claim 1, wherein the mobile device performs operations,further comprising: receiving user profile information about a user of asecond mobile device located in the orientational direction of themobile device; and refining the search request using the user profileinformation.
 5. The apparatus of claim 4, wherein when the user of thesecond mobile device indicates that results of the user is private,preventing display of information about the user at the mobile device.6. The apparatus of claim 1, wherein performing the search requestfurther comprises using a distance parameter to limit the searchresults.
 7. A system, comprising: an imaging device; and a processorwithin a mobile device, the processor is enabled to perform actions,including: determining a geographic position of the mobile device;determining a change in an orientational direction of the mobile devicefrom a first direction to a second direction, the orientationaldirection being determined at least in part using the imaging device ofthe mobile device; receiving one or more search terms at the mobiledevice; sending a search request using the received geographic position,change in orientational direction of the mobile device, and the receivedsearch terms; and displaying results from the search request on adisplay device of the mobile device.
 8. The system of claim 7, whereinthe search request further employs information that is based on trackingactions of the mobile device.
 9. The system of claim 7, wherein at leastone search term includes information about a type of merchant.
 10. Thesystem of claim 9, wherein the type of merchant is obtained using aselectable tab within a display on the mobile device.
 11. The system ofclaim 7, wherein the search results are limited based on a distance fromthe mobile device.
 12. The system of claim 7, wherein at least onesearch term includes information about a person.
 13. The system of claim12, wherein at least one search term further includes presenceinformation about the person.
 14. A computer-based method operatingwithin a mobile device, comprising: determining a geographic position ofthe mobile device; determining a change in an orientational direction ofthe mobile device from a first direction to a second direction, theorientational direction being determined at least in part using animaging component of the mobile device; receiving one or more searchterms at the mobile device; performing a search request using thereceived geographic position, change in orientational direction of themobile device, and the received search terms; and displaying results ofthe search request at the mobile device.
 15. The computer-based methodof claim 14, wherein at least one search term includes information abouta type of merchant in which to perform the search request.
 16. Thecomputer-based method of claim 14, wherein the search request isdirected towards providing information about a person within a defineddistance from the mobile device and within the orientational directionof the mobile device.
 17. The computer-based method of claim 14, whereinthe search request is refined based on tracked information about themobile device.
 18. The computer-based method of claim 14, wherein atleast one search term includes information about a merchant within adefined distance from the mobile device and orientational direction ofthe mobile device.
 19. The computer-based method of claim 14, whereindetermining a change in direction comprises analyzing a plurality ofimages captured by the imaging component to determine the orientationaldirection based on a shift between image patterns in the plurality ofimages.
 20. The computer-based method of claim 14, wherein the searchresults include information about a distance between the mobile deviceand at least one of a landmark, building, or an other computing device.